Experiments of Robotic Assembly Instructed by Situated Natural Language

Robots in the future are envisioned to become versatile helpers of man. Major challenges are to make them intelligent enough to enable exible and robust human-robot cooperation in shared workspaces and, moreover, to provide a simple, intuitive communication link between humans and robots. In Bielefeld, Germany, we follow an interdisciplinary approach involving linguistics, cognitive psychology, arti cial intelligence, and robotics research to tackle these challenges [2]. This video presents experimental results in using situated natural language for the instruction of robotic systems. First, the CODY Virtual Constructor demonstrates several examples of interactive assembly in a virtual environment. The basic scenario is the assembly of a toy airplane from the parts of a wooden construction kit. Since the changing situation in the task environment is dynamically conceptualized, not only single parts, but also constructed assembly groups, can be referenced in natural language instructions [3]. Second, a real robotic system demonstrates the construction of a sub-assembly of the toy airplane under instruction in natural language. This \bi-manual" robot consists of two cooperating manipulators, several cameras and force/torque sensors [1]. Visually supervised motion and sensor-based learning control enable robust performance of elementary operations like grasping, insertion and screwing [4].